Abstract

Model fit analysis

2 x 2 setzise by iteration(learn vs test) ANOVA table for behavioral data
term df sumsq meansq statistic p.value
setSize 1 0.0003843 0.0003843 0.0292196 0.8642903
iteration 1 7.7448344 7.7448344 588.7892899 0.0000000
setSize:iteration 1 0.6490973 0.6490973 49.3466348 0.0000000
Residuals 1988 26.1498145 0.0131538 NA NA
2 x 2 setzise by iteration(learn vs test) ANOVA table for model data
term df sumsq meansq statistic p.value
setSize 1 0.1612980 0.1612980 15.49667 8.55e-05
iteration 1 9.5609337 9.5609337 918.56458 0.00e+00
setSize:iteration 1 0.7419769 0.7419769 71.28527 0.00e+00
Residuals 1988 20.6922155 0.0104086 NA NA

RMSE vs Log-Likelihood: which method resulted in better a fit?

Correlation b/n behavioral and model data from the two procedures:

#> 
#>  Welch Two Sample t-test
#> 
#> data:  corr by method
#> t = 6.3954, df = 317.1, p-value = 5.735e-10
#> alternative hypothesis: true difference in means is not equal to 0
#> 95 percent confidence interval:
#>  0.04212052 0.07955197
#> sample estimates:
#> mean in group rmse mean in group logL 
#>          0.9046639          0.8438277
  • Looking at the violin plots above, data generated by the RMSE method is more strongly correlated (mean r: 0.9046639) with the behavioral data than those from the Log-Likelihood data (mean r: 0.8438277 ).

RMSE analysis

t-test on mean squared errors
estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high method alternative
0.0374 0.128 0.0905 8.61 0 149 0.0288 0.046 Welch Two Sample t-test two.sided
  • The log-likelihood method had lower errors (RMSE: 0.0904838) than the RMSE method (RMSE: 0.1278827) when comparing fits of the selected models to behavioral data by the two methods. This difference is significant as determined by a t-test.

BIC Analysis

Statistics on where the next best fit occurs for each participant by model
model mean median sd min max
biased 43.2459 8 119.99204 2 657
LTM 13.5000 9 15.20561 2 48
metaRL 3.0000 3 0.00000 3 3

These figures (specifically BIC fig 2) show that the best fit model has strong evidence against the second best fit model.These fits are better compared to the RMSE method (see RMSE document for comparison).

BIC fig 1

BIC fig 1

BIC Fig 2

BIC Fig 2

BIC fig 3

BIC fig 3

These subjects had second best fit models that came from a different model group. X1 to X3 are the BIC differences.
subjects X1 X2 X3 model
6218 1.6800691 0.7662433 0.0131273 LTM
6226 20.7272476 0.1642479 1.5106469 LTM
6234 17.4765377 0.4880064 0.3893591 LTM
15005 26.7061902 6.8416407 8.8128153 biased
15006 14.9020846 17.2172782 3.6532177 biased
15007 26.5035268 0.2165939 5.6548364 biased
15012 22.6166963 3.6827696 6.3863653 LTM
15020 21.0249436 7.4761139 4.9440946 LTM
28307 4.3697057 2.1578692 3.2079443 biased
29245 0.0178342 7.1627899 3.3654804 biased

Additional metrics

Figure 11.

Figure 11.

#> 
#>  Welch Two Sample t-test
#> 
#> data:  estimate by setSize
#> t = 10.149, df = 142.26, p-value < 2.2e-16
#> alternative hypothesis: true difference in means is not equal to 0
#> 95 percent confidence interval:
#>  0.02799973 0.04154511
#> sample estimates:
#> mean in group s3 mean in group s6 
#>       0.11481641       0.08004399
Descriptive stats of model and behavioral learning rate
setSize type model mean se
s3 behav biased 0.1163739 0.0021718
s3 behav LTM 0.1130357 0.0036576
s3 behav metaRL 0.0851190 0.0017857
s3 model biased 0.1141015 0.0032627
s3 model LTM 0.1131286 0.0012347
s3 model metaRL 0.0973333 0.0065714
s6 behav biased 0.0792870 0.0036193
s6 behav LTM 0.0826984 0.0043388
s6 behav metaRL 0.0765873 0.0115079
s6 model biased 0.0849828 0.0022839
s6 model LTM 0.1022738 0.0017649
s6 model metaRL 0.0829286 0.0030714
Figure 12

Figure 12

K-W test one way rank-sum test: s6-s3 learning curve differences by model type for behavioral data
statistic p.value parameter method
193.4056 0 2 Kruskal-Wallis rank sum test
K-W test one way rank-sum test: s6-s3 learning curve differences by model type for model data
statistic p.value parameter method
391.6311 0 2 Kruskal-Wallis rank sum test
pairwise post-hoc tests for behav data
group1 group2 p.value
LTM biased 0
metaRL biased 0
metaRL LTM 0
pairwise post-hoc tests for model data
group1 group2 p.value
LTM biased 0
metaRL biased 0
metaRL LTM 0
Figure 13.

Figure 13.

(2)set-size x (2)type(modelorBehav data) x 3(model) anova. RL excluded.
term df sumsq meansq statistic p.value
setSize 1 5.5580884 5.5580884 438.4466841 0.0000000
type 1 0.1911125 0.1911125 15.0757993 0.0001050
model 2 1.0508771 0.5254386 41.4489263 0.0000000
setSize:type 1 0.0062083 0.0062083 0.4897418 0.4840835
setSize:model 2 0.9516216 0.4758108 37.5340689 0.0000000
type:model 2 0.0831864 0.0415932 3.2810574 0.0376904
setSize:type:model 2 0.0648987 0.0324494 2.5597497 0.0774516
Residuals 3972 50.3521358 0.0126768 NA NA

Parameter analysis

mean and medians of parameter values
variable mean median
alpha 0.1500000 0.1500000
egs 0.3000000 0.3000000
bll 0.5000000 0.5000000
imag 0.3000000 0.3000000
ans 0.3000000 0.3000000
strtg 0.4402444 0.3604394
Figure 15

Figure 15

Individual Plots